Real Time Object Detection, Tracking, and Distance and Motion Estimation based on Deep Learning: Application to Smart Mobility

International audienceIn this paper, we will introduce our object detection, localization and tracking system for smart mobility applications like traffic road and railway environment. Firstly, an object detection and tracking approach was firstly carried out within two deep learning approaches: You Only Look Once (YOLO) V3 and Single Shot Detector (SSD). A comparison between the two methods allows us to identify their applicability in the traffic environment. Both the performances in road and in railway environments were evaluated. Secondly, object distance estimation based on Monodepth algorithm was developed. This model is trained on stereo images dataset but its inference uses monocular images. As the output data, we have a disparity map that we combine with the output of object detection. To validate our approach, we have tested two models with different backbones including VGG and ResNet used with two datasets : Cityscape and KITTI. As the last step of our approach, we have developed a new method-based SSD to analyse the behavior of pedestrian and vehicle by tracking their movements even in case of no detection on some images of a sequence. We have developed an algorithm based on the coordinates of the output bounding boxes of the SSD algorithm. The objective is to determine if the trajectory of a pedestrian or vehicle can lead to a dangerous situations. The whole of development is tested in real vehicle traffic conditions in Rouen city center, and with videos taken by embedded cameras along the Rouen tramway

TRAJECTOGRAPHY ESTIMATION FOR A SMART POWERED WHEELCHAIR ORB-SLAM2 VS RTAB-MAP A PREPRINT

International audienceThis work is part of the ADAPT project relating to the implementation of a trajectography functionality that aims to measure the path travelled by a patient during the clinical trials. This system (hardware and software) must be reliable, quickly integrable, low-cost and real-time. Therefore, our choices have been naturally made towards visual SLAM-based solutions coupled with an Intel real-sense consumer sensors. This paper is a comparison of two well-known visual SLAM algorithms in the scientific community, ORB-SLAM2 and RTAB-Map, evaluated in different path configurations. The added value of our work lies in the accurate estimation of the trajectories achieved through the use of a VICON motion capture system

Development in Vision-Based On-Road Behaviors Understanding

On-road behavior analysis is a crucial and challenging problem in the autonomous driving vision-based area. Several endeavors have been proposed to deal with different related tasks and it has gained wide attention recently. Much of the excitement about on-road behavior understanding has been the labor of advancement witnessed in the fields of computer vision, machine, and deep learning. Remarkable achievements have been made in the Road Behavior Understanding area over the last years

Real-Time 3D Multi-Object Detection and Localization Based on Deep Learning for Road and Railway Smart Mobility

International audienceFor smart mobility, autonomous vehicles, and advanced driver-assistance systems (ADASs), perception of the environment is an important task in scene analysis and understanding. Better perception of the environment allows for enhanced decision making, which, in turn, enables very high-precision actions. To this end, we introduce in this work a new real-time deep learning approach for 3D multi-object detection for smart mobility not only on roads, but also on railways. To obtain the 3D bounding boxes of the objects, we modified a proven real-time 2D detector, YOLOv3, to predict 3D object localization, object dimensions, and object orientation. Our method has been evaluated on KITTI’s road dataset as well as on our own hybrid virtual road/rail dataset acquired from the video game Grand Theft Auto (GTA) V. The evaluation of our method on these two datasets shows good accuracy, but more importantly that it can be used in real-time conditions, in road and rail traffic environments. Through our experimental results, we also show the importance of the accuracy of prediction of the regions of interest (RoIs) used in the estimation of 3D bounding box parameter

Environment Perception-based Deep Learning. Application to Road and Railway Smart Mobility.

International audienc

STAF: Spatio-Temporal Attention Framework for Understanding Road Agents Behaviors

International audienceOn-road behavior analysis is a key task required for robust autonomous vehicles. Unlike traditional perception tasks, this paper aims to achieve a high-level understanding of road agent activities. This allows better operation under challenging contexts to enable better interaction and decision-making in such complex environments. In this paper, we tackle the problem of discriminating spatio-temporal features that capture the visual instants that require more attention. We propose a new approach called STAF (Spatio-Temporal Attention Framework) through Long Short Term Memory (LSTM) layers that uses a multi-head attention mechanism on its past cell state to focus on attributes that are relevant over time. Experiments have been carried out on two different scenarios over data from Joint Attention in Autonomous Driving (JAAD) and Honda Research Institute Driving Dataset (HDD), both datasets devoted to understanding the behavior of road agents. The evaluation and results obtained proof that the proposed “STAF” is outperforming state-of-the-art algorithms-based LSTM (Ramanishka, Rasouli, and LSTM-baseline). For example, STAF outperforms LSTM with mean Average Precision (mAP) of 13%

Recent Advances in Vision-Based On-Road Behaviors Understanding: A Critical Survey

International audienceOn-road behavior analysis is a crucial and challenging problem in the autonomous driving vision-based area. Several endeavors have been proposed to deal with different related tasks and it has gained wide attention recently. Much of the excitement about on-road behavior understanding has been the labor of advancement witnessed in the fields of computer vision, machine, and deep learning. Remarkable achievements have been made in the Road Behavior Understanding area over the last years. This paper reviews 100+ papers of on-road behavior analysis related work in the light of the milestones achieved, spanning over the last 2 decades. This review paper provides the first attempt to draw smart mobility researchers’ attention to the road behavior understanding field and its potential impact on road safety to the whole road agents such as: drivers, pedestrians, stuffs, etc. To push for an holistic understanding, we investigate the complementary relationships between different elementary tasks that we define as the main components of road behavior understanding to achieve a comprehensive understanding of approaches and techniques. For this, five related topics have been covered in this review, including situational awareness, driver-road interaction, road scene understanding, trajectories forecast, driving activities, and status analysis. This paper also reviews the contribution of deep learning approaches and makes an in-depth analysis of recent benchmarks as well, with a specific taxonomy that can help stakeholders in selecting their best-fit architecture. We also finally provide a comprehensive discussion leading us to identify novel research directions some of which have been implemented and validated in our current smart mobility research work. This paper presents the first survey of road behavior understanding-related work without overlap with existing reviews

Self-Supervised Sidewalk Perception Using Fast Video Semantic Segmentation for Robotic Wheelchairs in Smart Mobility

International audienceThe real-time segmentation of sidewalk environments is critical to achieving autonomous navigation for robotic wheelchairs in urban territories. A robust and real-time video semantic segmentation offers an apt solution for advanced visual perception in such complex domains. The key to this proposition is to have a method with lightweight flow estimations and reliable feature extractions. We address this by selecting an approach based on recent trends in video segmentation. Although these approaches demonstrate efficient and cost-effective segmentation performance in cross-domain implementations, they require additional procedures to put their striking characteristics into practical use. We use our method for developing a visual perception technique to perform in urban sidewalk environments for the robotic wheelchair. We generate a collection of synthetic scenes in a blending target distribution to train and validate our approach. Experimental results show that our method improves prediction accuracy on our benchmark with tolerable loss of speed and without additional overhead. Overall, our technique serves as a reference to transfer and develop perception algorithms for any cross-domain visual perception applications with less downtime

Toward Comprehensive Road Agents BehaviorUnderstanding

International audienc

Estimation de trajectoire pour un fauteuil roulant électrique

International audienc